Product identification systems and methods

ABSTRACT

A system for identifying a product removed from a cabinet includes detecting visual characteristics of the product removed from the cabinet by a camera within the cabinet. The system may include detecting an identifier on the product removed from the cabinet by an identifier sensor, and comparing the visual characteristics and the identifier to a database of product information. The product removed from the cabinet may be identified based on the comparison of the visual characteristics and the identifier to the database of product information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/864,676, filed May 1, 2020, which is incorporated herein by referencein its entirety.

FIELD

Embodiments described herein generally relate to systems and methods foridentifying products. Specifically, embodiments described herein relateto systems and methods for identifying products removed from anunattended vending machine by a consumer.

BACKGROUND

Vending machines generally require a consumer to enter a payment, make aproduct selection, and wait for the product to be dispensed by thevending machine. However, the consumer can encounter multiple problemswhen using a vending machine. First, the vending machine may not acceptthe consumer's form of payment. For example, the vending machine may notaccept paper bills that are creased or wrinkled. The vending machine maynot properly register receipt of bills or coins, and thus the consumermay not receive credit for entered payment. The vending machine may notbe configured to accept mobile payment which can be inconvenient for theconsumer. Further, the vending machine may fail to read a payment card,such as a credit card or debit card. As a result, the consumer may beunable to make a purchase, or the consumer may become frustrated anddecide to not use the vending machine.

Second, once payment is entered, the consumer may incorrectly enter thecode corresponding to the desired product. As a result, a differentproduct may be dispensed than the consumer anticipated. The consumer maybe unable to return the incorrect product, and the consumer may have norecourse. Further, vending machines generally allow a consumer topurchase only a single product at a time, requiring the consumer torepeat the process of entering payment and selecting a product in orderto purchase multiple products. Repeating the same steps can be timeconsuming and frustrating, and may deter the consumer from makingmultiple purchases.

Third, the vending machine may fail to properly convey the selectedproduct to the user. For example, a screw drive may fail to move theproduct to the dispensing opening of the vending machine, or a gateholding the product in place may not fully open. Additionally, theproduct may become stuck or lodged within the vending machine and maynot be accessible by the consumer. As a result, the consumer may notreceive the product and cannot obtain a refund of their payment.

Vending machines have various additional drawbacks such as the inabilityfor the consumer to personally select a specific product. Instead, theconsumer simply selects the type of product, but cannot pick the exactproduct to be dispensed. Further, the consumer cannot handle or inspectthe product before purchasing. As a result, the consumer may not be ableto learn about the product, such as to read the label, ingredients, ornutritional information. This may discourage the consumer frompurchasing products that are not familiar. The dispensed product may bedamaged, expired, or otherwise deficient. These various factors cancontribute to a poor consumer experience.

Thus, improved vending machines are desired that provide a simple andeasy purchasing experience. Further, vending machines are desired thatallow a consumer to personally select one or more products and thatensure dispensing of the desired product.

BRIEF SUMMARY OF THE INVENTION

Some embodiments described herein relate to a method for identifying aproduct removed from a cabinet, the method including detecting visualcharacteristics of the product removed from the cabinet by a camerawithin the cabinet, detecting an identifier on the product removed fromthe cabinet by an identifier sensor, comparing the visualcharacteristics and the identifier to a database of product information,and identifying the product removed from the cabinet based on thecomparison of the visual characteristics and the identifier to thedatabase of product information.

Some embodiments described herein relate to a method for identifying aproduct removed from a cabinet, the method including capturing a firstimage of a plurality of products within the cabinet by an internalcamera within the cabinet, removing a product of the plurality ofproducts from the cabinet, capturing a second image of the plurality ofproducts within the cabinet by the internal camera after removing theproduct, determining an identity of the product removed from the cabinetby analyzing the first image and the second image, and confirming theidentity of the product removed from the cabinet by detecting anidentifier of the product removed from the cabinet.

Some embodiments described herein relate to a method for identifyingproducts removed from a cabinet, the method including detecting alocation from which the product was removed from the cabinet via atleast one of a camera and a sensor, determining a visual characteristicof the product based on data from the at least one of a camera and asensor, determining a predicted identity of the product based on thevisual characteristic, determining an identity of the product at thelocation based on a digital map of products within the cabinet, andconfirming that the predicted identity of the product corresponds to theidentity of the product based on the digital map.

In any of the various embodiments discussed herein, the visualcharacteristics may include a shape of the product.

In any of the various embodiments discussed herein, the visualcharacteristics may include a coloring of the product.

In any of the various embodiments discussed herein, the identifiersensor may be the camera.

In any of the various embodiments discussed herein, the identifier maybe a barcode.

In any of the various embodiments discussed herein, the method mayfurther include confirming the identity of the product by determining aweight of the product removed from the cabinet with a weight sensorarranged within the cabinet. In some embodiments, determining the weightof the product removed from the cabinet may include determining a firstweight of products in the cabinet via the weight sensor, determining asecond weight of products in the cabinet via the weight sensor afterremoving the product from the cabinet, and calculating a differencebetween the first weight and the second weight.

In any of the various embodiments discussed herein, the method mayfurther include detecting a visual characteristic of the product removedfrom the cabinet at a location outside of the cabinet by an externalcamera, and confirming the identity of the product removed based on thevisual characteristic detected by the external camera.

In any of the various embodiments discussed herein, the method mayfurther include detecting a user by an external camera, detecting by theexternal camera unauthorized conduct by the user, and locking thecabinet when the unauthorized conduct is detected.

In any of the various embodiments discussed herein, the method mayfurther include detecting data relating to the product removed via anoptical sensor, and confirming the identity of the product removed usingthe data from the optical sensor.

In any of the various embodiments discussed herein, the optical sensormay include a

LIDAR sensor.

In any of the various embodiments discussed herein, the method mayfurther include updating the digital map of products within the cabinetafter the product is removed from the cabinet.

In any of the various embodiments discussed herein, the method mayfurther include generating the digital map of the products within thecabinet using data received from the at least one of a camera and asensor.

In any of the various embodiments discussed herein, the digital map mayinclude a location and a model of each product within the cabinet.

In any of the various embodiments discussed herein, the method mayfurther include detecting an identifier of the product removed from thecabinet via an identifier sensor.

BRIEF DESCRIPTION OF THE DRAWINGS/FIGURES

The accompanying drawings, which are incorporated herein and form a partof the specification, illustrate the present disclosure and, togetherwith the description, further serve to explain the principles thereofand to enable a person skilled in the pertinent art to make and use thesame.

FIG. 1 shows a front view of a vending machine configured to identifyproducts removed from the cabinet according to an embodiment.

FIG. 2 shows a schematic diagram of components of a system foridentifying a product removed from a cabinet of a vending machineaccording to an embodiment.

FIG. 3 shows a top-down view of a shelf of a vending machine showing anarrangement of cameras according to an embodiment.

FIG. 4 shows a top-down view of a shelf of a vending machine showing thelocation of cameras according to an embodiment.

FIG. 5 shows a perspective view of a vending machine having camerasaccording to an embodiment.

FIG. 6 shows an exemplary method of determining an identity of a productaccording to an embodiment.

FIG. 7 shows an exemplary method of determining an identity of a productaccording to an embodiment.

FIG. 8 shows a side view of a vending machine having a weight sensoraccording to an embodiment.

FIG. 9 shows an exemplary method of determining an identity of a productaccording to an embodiment.

FIG. 10 shows an exemplary method of determining whether a product hasbeen returned to the cabinet according to an embodiment.

FIG. 11 shows a view of products in a shelf as determined by an opticalsensor according to an embodiment.

FIG. 12 shows a perspective view of a cabinet of a vending machine and adigital map of the products according to an embodiment.

FIG. 13 shows a top-down view of a system for identifying a product bytracking a location of a consumer's hand according to an embodiment.

FIG. 14 shows an exemplary method of determining an identity of aproduct according to an embodiment.

FIG. 15 shows a schematic block diagram of an exemplary computer systemin which embodiments may be implemented.

DETAILED DESCRIPTION OF THE INVENTION

Reference will now be made in detail to representative embodimentsillustrated in the accompanying drawings. It should be understood thatthe following descriptions are not intended to limit the embodiments toone preferred embodiment. To the contrary, it is intended to coveralternatives, modifications, and equivalents as can be included withinthe spirit and scope of the described embodiments as defined by theclaims.

Some vending machines may provide consumers with access to thecompartment in which the products are stored. In this way, the consumercan inspect the products to review the label, nutritional informationand the like when deciding whether to purchase the product. Further, theconsumer can select the exact product desired to be purchased. Theconsumer may readily purchase multiple products in a single transaction.

Such vending machines may accept a payment source from a consumer oridentify the consumer, provide the consumer with access to the cabinetin which the products are stored, detect the products removed from thecabinet by the consumer, and charge the consumer for the selectedproducts. While such vending machines may provide added convenience tothe consumer, accurately detecting products selected by the consumerpresents numerous technical challenges. If products removed from thecabinet are not identified and charged to the consumer, the owner of thevending machine may lose income. Additionally, if products removed arenot correctly identified, the consumer may be charged the wrong price,and the inventory of the vending machine may be incorrectly maintained.Considerations must also be made to ensure that consumers do not tamperwith products or otherwise engage in unauthorized activities.

In order to ensure accurate identification of products removed from thevending machine, the vending machine must be able to differentiatebetween a variety of products, many of which may be similar inappearance. For example, many beverage bottles may be the same orsimilar in size and shape, particularly beverages from the samemanufacturer. Thus, some products may differ only by small details onthe packaging, such as the product name or coloring.

The vending machine must also be able to detect the product in variousorientations. The products may be arranged in the cabinet with variousorientations, and consumers may remove products from the vending machinein different manners. Consumers may select products in ways that obscurethe product, inhibiting identification. For example, a consumer may grabmultiple products in one hand, making it difficult to detect eachindividual product selected. Consumers may also remove a product andreturn the product to a different location and in a differentorientation than the product was initially positioned.

In order to ensure proper use of the vending machine, precautions mustbe taken to prevent unauthorized conduct, such as stealing or tamperingwith products, or damaging the vending machine. A consumer may try toremove products without detection so that the product will not becharged to the consumer. Alternatively, a consumer may try to deceivethe vending machine by inserting external objects into the vendingmachine in place of products to make it appear that a product has beenreturned. If consumers are able to steal or tamper with products, theowner of the vending machine may suffer loss of income. Other consumersmay not choose to use the vending machine if the products available forpurchase are damaged.

Some embodiments described herein relate to systems and methods foridentifying products removed from a cabinet using a camera and anidentifier sensor in which data collected by the camera and identifiersensor are compared to a database of product information. In this way, aproduct can be accurately identified without the consumer having tomanually scan or enter information about a product, simplifying purchaseof products from the vending machine. Some embodiments described hereinrelate to systems and methods for identifying a product removed from acabinet that includes generating a digital map of products within thecabinet. The digital map provides a baseline of the location andidentity of products within the cabinet and may be used to confirmidentification of a product removed from the cabinet by cameras orsensors in the cabinet.

In some embodiments, a vending machine 100 may include a cabinet 110having a plurality of products 200 stored within cabinet 110, as shownin FIG. 1 . Cabinet 110 may further include a door 118 that can beopened to provide access to the plurality of products 200 within cabinet110. Cabinet 110 may include one or more cameras 120, 130 or sensors140, 150, 160 for identifying products. Specifically, cameras andsensors are configured to identify products removed from cabinet 110 sothat the consumer is charged only for products removed from cabinet 110and not returned to cabinet 110. Data collected by the sensors andcameras is analyzed, such as by a control unit 180 (see FIG. 2 ), todetermine the identity of the product removed. The analysis may includea comparison of the data from the cameras and sensors to a database ofproduct information and/or a product inventory.

The detection system and methods described herein may be used in avending machine that allows a user to manually select and removeproducts from a cabinet in which the products are stored. A vendingmachine that allows a consumer to manually select and remove products isdescribed for example in U.S. application Ser. No. 16/559,300, filedSep. 3, 2019, incorporated herein by reference in its entirety. Anexemplary vending machine incorporating a product identification systemand method is described herein for illustrative purposes only. One ofordinary skill in the art will appreciate that the productidentification system and methods described herein can be used withother types of vending machines or merchandisers, and can be utilized inother environments for product identification.

A vending machine 100 may have components as shown for example in FIG. 2. However, vending machine 100 need not have each and every componentshown in FIG. 2 , and may include additional components.

Vending machine 100 may be configured to authenticate a consumer'sidentity. Vending machine 100 may include an external camera 130 toidentify a consumer by facial recognition, or a biometric sensor 172 toobtain biometric information from the consumer, such as a thumbprint oriris. In some embodiments, vending machine 100 may alternatively oradditionally include a communication device 174, such as a wirelesstransceiver, for communicating with a mobile device, such as a cellphone, so that the consumer may authenticate or provide payment via amobile device. In such embodiments, the mobile device may have asoftware application to facilitate interaction with vending machine 100.The consumer's identity may be linked with a consumer profile thatincludes information about the consumer, such as a payment source, sothat the consumer need not manually enter a payment when using vendingmachine 100, and the consumer's purchase can be automatically creditedto the consumer's profile.

Vending machine 100 may not require authentication of a consumer and maysimply accept a form of payment from the consumer. Vending machine 100may include a payment processing unit 170 that may include one or moreslots to receive paper money, coins, or tokens. Payment processing unit170 may include a card reader to read a magnetic stripe or an electronicchip of a credit card, debit card, gift card, or the like, or thatincludes a near field communication (NFC) antenna to receive contactlesspayment from a contactless payment card. Payment processing unit 170 mayinclude a communication device to accept mobile payments orcryptocurrency from a mobile electronic device, such as a cell phone,watch, laptop, tablet, or the like, or payment processing unit 170 mayinclude a scanner to scan a payment code, such as a quick response (QR)code.

Upon authenticating a consumer's identity or receiving a payment fromthe consumer, door 118 of vending machine 100 may be automaticallyunlocked so that the consumer may access the plurality of products 200.The products removed by the consumer may be identified by methods asdescribed herein. A virtual shopping cart displayed on a user interface176 or on the user's mobile device may list the products removed fromcabinet 110 along with the price of the products, and a total price ofthe products.

Purchase of the removed products may be completed when the consumercloses door 118 of vending machine 100. To complete the purchase, theconsumer may provide an input, such as making a selection to completethe transaction on a user interface 176 of vending machine 100, orperforming a gesture on a user interface 176 having a touch screen, suchas swiping along a path. Alternatively, the purchase may be completedautomatically when door 118 is closed for a predetermined period oftime.

In some embodiments, vending machine 100 may include one or moreinternal cameras 120 within cabinet 110 for identifying products, asshown in FIG. 3 . Cameras 120 may be configured to capture staticimages, cameras 120 may capture videos, or both. In some embodiments, aplurality of cameras 120 may be arranged to capture images or videos ofproducts on each shelf 112 of cabinet 110. A camera 120 may be arrangedat one or more corners of cabinet 110 above each shelf 112. Further, insome embodiments, a camera 120 may be arranged centrally above eachshelf 112 so as to capture an image of a central portion of shelf 112.For example, a camera 120 may be arranged in each of four corners ofcabinet 110 and a fifth camera may be arranged above a central portionof shelf 112. In this way, the cameras 120 may capture products fromdifferent angles and can detect products that may be obscured in a viewfrom a particular camera 120. Further, cameras 120 can view any productswithin cabinet 110, and may have overlapping fields of view. Forexample, a camera 120 arranged at a front portion 116 of cabinet 110 maynot fully capture a product arranged at a back portion 114 of cabinet110. A best image from cameras 120 may be selected for analysis todetermine the identity of the product, or a composite imageincorporating various images may be generated and analyzed.

In some embodiments, one or more cameras 120 may be configured tocapture images or video of a product exiting (or entering) the cabinet110, as shown in FIG. 4 . Cameras 120 may be arranged at front portion116 of cabinet 110. In some embodiments, cameras 120 may be positionedat corners of front portion 116 of cabinet 110 or may be positionedabout a perimeter of front portion 116 of cabinet 110. Cameras 120 maydefine a plane P that is parallel to a front portion 116 of cabinet 110.In this way, a product 200B exiting or entering the cabinet 110 mustpass through plane P and is thus detected by cameras 120. In suchembodiments, products 200A within cabinet 110 may not be detected bycameras 120. Cameras 120 may capture an image or video to identifyproduct 200B removed from cabinet 110 or returned to cabinet 110.Cameras 120 at front portion 116 of cabinet 110 may have a clear view ofa product 200 as it is being removed by the consumer as the product 200is not obscured by other products within the cabinet.

Cameras 120 may be used to detect a visual characteristic of a product.The visual characteristic may include a shape of the product, adimension of the product, a coloring of the product, or a combinationthereof. Cameras may also be used to determine a location of a productwithin cabinet 110.

The shape of the product may be a silhouette or 2-D view of the product,such as a front profile, a side profile, a rear profile, a top-downview, or a bottom-up view. For example, if the product is a can, theshape may be a circular shape when viewed in a top-down manner, or agenerally rectangular shape when viewed in a side profile. In someembodiments, shape may be a 3-D view, such as a perspective view of theproduct. The 3-D view may be generated by combining the 2-D views fromvarious cameras. In some embodiments, cameras may be used to generate amodel of each product. The model may be a 2-D model that includes ashape and color or color palette. In some embodiments, the model may bea 3-D model that includes the product's shape, dimensions, and a coloror color palette. Cameras 120 may have depth sensors to aid ingeneration of the 3-D model. The cameras may determine the dimensions ofthe product so that products with similar shapes may be distinguished.For example, a 12 oz. can and a 16 oz. can are distinguishable despiteboth being cylindrical. In some embodiments, in order to ensureaccuracy, cameras may be configured to determine the dimensions ofproducts within ±5 mm, ±3 mm, or ±1 mm.

The visual characteristic may include a coloring of the product. Thecoloring may be a color of any portion of the product, or a pattern orcombination of colors, e.g., a color palette. For example, the visualcharacteristic may be the color of the packaging, the color of text,logos or markings on the packaging, among other colored items. Forexample, when the product is a bottled beverage, the coloring may be acolor of the bottle (e.g., clear, green), a color of the liquid withinthe bottle, a color of the bottle cap, a color of the label, or a colorof the writing or markings on the label, and combinations thereof.

In some embodiments, an identifier sensor 150 (see, e.g., FIG. 1 ) maydetect an identifier 210 of a product 200. An identifier 210 of aproduct may include a label, barcode, QR code, text (such as a brand,product, or flavor name), logo, or other markings on the product. Insome embodiments, the identifier sensor may be a camera 120. In someembodiments, identifier sensor 150 may be a separate component, such asa scanner for scanning a barcode or QR code. Identifier sensor 150 mayhave sufficiently high resolution so that text on a product 200 can beread. In some embodiments, a control unit 180 of vending machine 100 mayperform optical character recognition (OCR) to identify text in acaptured image of a product. Captured images or video may havesufficient pixel density to allow for accurate identification of thetext. In some embodiments, a minimum pixel density for identifying textmay be about 2 pixels/mm. Further, identifier sensor 150 may have a highframe rate to provide sharp images to facilitate OCR.

In some embodiments, a convolutional neural network (CNN) may be used todetect an identifier 210 on a product 200 as will be appreciated by oneskilled in the art, such that the identifier 210 may be analyzed forproduct recognition. CNN may be trained based on products available invending machine 100 to increase accuracy. Further, identifier sensor 150may have sufficient resolution to resolve differences in identifiers 210of related products (e.g., Pepsi, diet Pepsi, cherry Pepsi). In someembodiments, for example, accurate identification of products mayrequire a minimum pixel density of 1.5 pixels/mm.

Identifier sensor 150 may assist in determination of the specific stockkeeping unit (SKU). For example, one or more cameras 120 may detect asize and shape of a product, but multiple products in cabinet 110 may bethe same size and shape. Thus, identifier sensor 150 may help todetermine the specific type of product by detecting an identifier 210 ofthe product. Alternatively, if the cameras 120 alone are able todetermine an identity of the removed product, the information providedby identifier sensor 150 may be used to increase confidence that theproduct has been correctly identified or to confirm that identificationof the product based on the cameras 120 is correct.

A control unit 180 may be configured to receive and analyze data fromthe cameras 120 and identifier sensor 150 to determine a productidentity. Control unit 180 may also store a database of productinformation. The database may include information about the productsstored in cabinet 110. The database may include for example a list ofproducts. For each product, the database may include correspondingvisual characteristics, such as a shape or silhouette, dimensions, andcoloring of the packaging, product weight, and further information abouta product label and identifiers. To identify a product removed from thecabinet, the analysis may determine a product in the database that hasvisual characteristics that correspond to, or best match, the visualcharacteristics determined based on data from cameras 120 and sensor150. In some embodiments, control unit 180 may execute sensor fusionalgorithms for determining product identity based on data from cameras120 and sensors 150. Artificial intelligence and machine learning may beused to analyze the data from cameras 120 and sensors in combinationwith the database of product information to determine a productidentity. In some embodiments, artificial intelligence may assign aconfidence level to the product identification. Computer visiontechnology may be used to analyze data, such as images or video fromcameras and sensors as will be understood by one of ordinary skill inthe art. In some embodiments, artificial intelligence or computer visiontechnology may be employed remotely from vending machine 100. Forexample, cloud computing, edge computing, or a combination thereof maybe used to analyze data from cameras 120 and sensors.

In some embodiments, control unit 180 may also store a product inventoryof vending machine 100 so that it is known what products are in cabinet110. Thus, the identification of products removed is limited to productsknown to be in cabinet 110, or on a particular shelf 112 from which theproduct 200 was removed. In some embodiments, control unit 180 maygenerate and store a digital map of products in cabinet 110 which mayfurther aid in product identification, as discussed in further detailbelow with respect to FIG. 12 .

In some embodiments, a vending machine 100 may include internal cameras120, as shown in FIG. 5 . Vending machine 100 includes a cabinet 110having shelves 112 on which products may be stored. A first plurality ofcameras 120B may be positioned above each shelf 112 so as to detectproducts on each shelf 112 inside of the cabinet 110. First plurality ofcameras 120B may include cameras 120B at corners of each shelf 112 andat a central portion of each shelf 112. A second plurality of cameras120A may be arranged at front portion 116 of cabinet 110. Secondplurality of cameras 120A may be configured to detect a product removedfrom or returned to cabinet 110, as described above with respect to FIG.4 . The second plurality of cameras 120A may be positioned about aperimeter of front portion 116. However, in some embodiments, fewer oradditional cameras 120 may be included.

In some embodiments, a method of determining an identity of a productremoved from a cabinet may include using a camera to capture a video ofproducts entering or exiting cabinet 600, as shown for example in FIG. 6. Cameras may be activated when the door of the cabinet is opened andmay deactivate when the door is closed so that video is captured onlywhen the cabinet is being accessed by a consumer. Camera 610 may detecta visual characteristic of a product removed from the cabinet. Anidentifier of the product removed from the cabinet may be detected by anidentifier sensor 620. The visual characteristic and identifier may beanalyzed to determine a product identity, which may be based on adatabase of product information 630. In some embodiments, the visualcharacteristic and identifier may be analyzed for correspondence with aproduct inventory or digital map of products in the cabinet. Theidentity of the product may be determined based on the analysis of thevisual characteristic and identifier and the database of productinformation 640. In some embodiments, the visual characteristic oridentifier alone may be used to identify the product, and the other ofthe visual characteristic or identifier may be used to confirm theidentity of the product.

In some embodiments, a method of determining an identity of a product700 may include the use of cameras, as shown in FIG. 7 . The cameras maycapture images at a set interval or cameras may capture an image whenthe door of the cabinet is closed. When multiple cameras are used, theimages from the multiple cameras may be combined into a composite image,or the best image may be used. The method may include capturing a firstimage of the plurality of products in the cabinet 710. The first imagemay be the baseline image showing the products before a consumeraccesses the cabinet. A consumer may remove one or more products fromthe cabinet 720. The camera may then capture a second image of theplurality of products within the cabinet 730. The identity of theproduct may be determined in part by analyzing the first and secondimages to determine what products have been removed 740. The analysis ofthe images may be used to determine a visual characteristic of theproduct removed, and/or a location from which the product was removed.The analysis may include comparison of the visual characteristic to adatabase of product information, and the product location may be used todetermine a product known to be stored at that location. Further, toconfirm the identity of the product removed, an identifier of theproduct removed may be detected by an identifier sensor 750. Bydetecting an identifier of the product, the accuracy of the productidentification may be increased relative to using data from the camerasalone.

In some embodiments, a combination of images and videos captured bycameras may be used to identify a product. In such embodiments, a cameramay capture an image of products within the cabinet. Another camera maycapture a video of a product being removed from cabinet. A second imagemay be captured of the products in the cabinet after the products havebeen removed. The video may be analyzed to determine a visualcharacteristic of the product, and artificial intelligence may use thevisual characteristic and a database of product information to determinethe identity of the product. The identity of the product as determinedbased on the captured video may be confirmed by an analysis of the firstand second images to determine a location or visual characteristic ofthe removed product. Alternatively, the product identification may bemade by analyzing the first and second images, and the data from thevideo may be used to confirm the identification.

In some embodiments, the identification of the product may be determinedin part using a weight sensor 140 (see, e.g., FIG. 1 ). Weight sensor140 may be configured to determine a weight of products in cabinet 110.In some embodiments, weight sensor 140 may be one or more load cells.Weight sensor 140 may be located on a shelf 112, or inside of a shelf112, such that weight sensors 140 may determine a weight of a productplaced on the shelf 112.

In some embodiments, weight sensor 140 may be arranged as shown in FIG.8 . Each shelf 112 within cabinet 110 of vending machine 100 may besupported on one or more weight sensors 140. In some embodiments, aweight sensor 140 is arranged at each corner of a rectangular shelf 112.Weight sensors 140 may be arranged on brackets 111 within cabinet 110.Thus, weight sensors 140 can detect the weight of shelf 112 and theproducts thereon to determine whether products have been removed from orreturned to a shelf 112.

An exemplary method of determining an identity of a product 900 using aweight sensor is shown in FIG. 9 . A first weight of products on a shelfmay be determined by a weight sensor 910. Products may be removed fromthe shelf by a consumer 920. A second weight of products on the shelfmay be determined by the weight sensor 930. The difference between thefirst weight and the second weight may be calculated to determine theweight of the product or products removed. The calculated weight may beanalyzed to determine a product or products in the database of productinformation that correspond to, or best match, the calculated weight ofthe product removed. The weight of the product removed can be used toconfirm the identity of the product removed that was determined based onthe cameras or identifier sensor.

In some embodiments, weight sensor 140 may also be used to provideinformation about the location of products in cabinet 110. A shelf 112in cabinet 110 may include multiple weight sensors 140. Thus, weightsensors 140 may help to indicate the location from which the product isremoved depending on which weight sensor 140 the product is placed. Themore weight sensors 140 included in cabinet 110, the greater the abilityof weight sensors 140 to determine the precise location of a product.Further, if the consumer returns a product, weight sensor 140 may helpto determine the location at which the returned product is placed. Thisinformation may be used to update a digital map of products withincabinet 110. Weight sensor 140 may also detect a product that has fallenover or that is obscured by other products and may not be readily viewedby cameras 120.

In some embodiments, a weight sensor may be used to determineunauthorized conduct by the consumer, as shown in FIG. 10 . In suchembodiments, a weight of a product removed may be determined by a weightsensor 1010, as discussed above with respect to FIGS. 8-9 . Weightsensor may also determine the weight of a product returned to thecabinet 1020. For example, a consumer may take out a product to read thelabel or nutritional information and may decide against purchasing theproduct and return the product to the cabinet. The weight of the productremoved is compared to the weight of the product returned 1030. If theweight of the product returned is the same as the weight of the productremoved, the return is accepted 1040. If the weight of the productreturned is different from the weight of the product returned, thereturn is not accepted 1050. Thus, the consumer may be charged for anyproducts removed and not returned. If the consumer attempts to deceivethe system by returning an external item in place of the product removedfrom the cabinet, the system can detect that the weight of the externalproduct differs from the weight of the product removed. Further, if theconsumer samples the product, such as by consuming a portion of theproduct, the weight of the product will decrease, and the return willnot be accepted.

In some embodiments, the determination of whether a product is properlyreturned can be aided by cameras 120. Cameras 120 may detect a visualcharacteristic of the product removed and of the product returned todetermine if the visual characteristic is the same. If the visualcharacteristic of the product returned differs from the visualcharacteristic of the product removed, the consumer may have tamperedwith the product or attempted to return a different item.

In some embodiments, an optical sensor 160 may be used to determine avisual characteristic and/or a location of a product (see, e.g., FIG. 1). Optical sensor 160 may be arranged within cabinet 110 and may viewsubstantially an entire of interior of cabinet. Optical sensor 160 mayuse different optical wavelengths. In some embodiments, optical sensor160 may be used to aid in determination of the location, size, and shapeof each object in cabinet 110, as shown for example in FIG. 11 . Datafrom optical sensors may determine the size, shape and location ofproducts that may be obscured from the view of cameras. Further, if aconsumer removes and returns a product to a different location, opticalsensor can determine the location of the product.

In some embodiments, optical sensor may be an RFID sensor. In suchembodiments, cabinet 110 may include an RFID sensor configured to detectthe presence of RFID tagged products. Thus, when a product is removedfrom cabinet, RFID sensor may determine the identity of the productremoved. In some embodiments, optical sensor may be a light detectionand ranging (LIDAR) sensor or a magnetic resonance imaging (Mill)sensor, among others. Data from optical sensor 160 may be used toconfirm the identity of the product removed from the cabinet asdetermined by other sensors or cameras. This may help to increase theaccuracy of the product identification.

In some embodiments, vending machine may generate a digital map ofproducts in cabinet, as shown in FIG. 12 . Digital map 300 may include adigital representation of cabinet and products therein. Thus, digitalmap 300 may be a real-time, digital twin of cabinet 110 and products incabinet 110. For example, as shown in FIG. 12 , a shelf 112 of cabinet110 may include products 201-204 arranged on shelf 112 in specificlocations. Digital map 300 may include a representation of cabinet andshelf 312, and may include models, such as 3D models of products201′-204′ on shelf 312. Thus, digital map 300 may include the locationand identity of products in cabinet 110 of vending machine 100. Digitalmap 300 may also serve as a product inventory so that the number of eachproduct in the cabinet is known.

In some embodiments, cameras 120, sensors, or a combination thereof maybe used to generate digital map 300. For example, as an operator placeseach product in cabinet 110, cameras 120 and sensors detect the locationand identity of the product, and may further generate a 3D model of theproduct. In this way, the digital map may be generated as the cabinet110 is filled. In some embodiments, an operator can manually input orconfirm the identity and location of each product within the cabinet.

Digital map 300 may provide a baseline of information about products inthe cabinet before a consumer accesses the cabinet. If a consumerremoves a product from a particular location within cabinet, theidentity of the product at that location is known from the digital map.Thus, the digital map may be used to confirm a product identificationmade based on data from cameras 120 or other sensors of vending machine100. Further, as consumers remove products from the cabinet, data fromcameras 120 and sensors may be used to update the digital map. Forexample, the digital map may be updated to reflect that one or moreproducts have been removed, one or more products have been restocked, ormay be updated to reflect that one or more products have been moved orrelocated within cabinet. Optical sensor 160 may help to determine alocation of products within cabinet as view of some products withincabinet may be obscured from view of cameras.

In some embodiments, a product identity may be determined in part bytracking a location of a consumer's hand, as shown in FIG. 13 . Thelocation of the consumer's hand may be tracked by computer-visiontechnology based on data from cameras 120. The coordinates of theconsumer's hand 1200 within cabinet 110 may be detected by one or morecameras 120 to determine which shelf the consumer is accessing. In someembodiments, consumer's hand 1200 may be tracked to determine a product200 at the location of the consumer's hand. The coordinates of theconsumer's hand may be determined in two-dimensions, such as in a topdown view of the shelf. For example, the coordinates of the consumer'shand 1200 may include a position along an X-axis that extends in adirection from a front to a rear portion of cabinet 110, and a positionalong a transverse axis, or Y-axis. The identity and location of eachproduct 200 may be known, such as by the digital map. Thus, if thelocation of the consumer's hand is known, the product 200 at thecoordinates of the consumer's hand 1200 can readily be identified bydetermining the product at that location in the digital map. Computervision may further detect a particular movement or gesture by theconsumer's hand 1200 to determine whether the consumer is picking up orreturning a product. In some embodiments, a trajectory of the consumer'shand 1200 may be detected to determine a product selected by theconsumer.

An exemplary method of determining a product identity using a digitalmap 1400 is shown in FIG. 14 . A location from which a product wasremoved from the cabinet may be detected by at least one of a camera andsensor 1410. A visual characteristic of the product removed from thecabinet may be determined based on data from the camera or sensor 1420.A predicted identity of the product at the location may be determinedbased on the visual characteristic 1430. For example, artificialintelligence and machine learning technology may be used to predict anidentity of the product based on a pre-trained machine learning model.For example, the visual characteristic may be a shape of the product,and the artificial intelligence may analyze products in the productinventory having known shapes that best match the shape of the product.In order to confirm that the predicted identity is correct, an identityof the product at the location may be determined based on a digital mapof products within the cabinet 1440. The predicted identity may beconfirmed if the predicted identity corresponds to the identity of theproduct based on the digital map 1450.

In some embodiments, vending machine 100 may include an external camera130, as shown in FIG. 1 . External camera 130 may be configured to viewan area external to vending machine 100. External camera 130 may bepositioned outside of cabinet 110, or may be arranged within cabinet 110so as to view an area external to vending machine 100. For example,external camera 130 may be arranged on a door of vending machine 100, onan exterior of cabinet 110, or camera 130 may be separate from vendingmachine 100. In some embodiments, external camera 130 may be activatedwhen a presence of a consumer is detected near the vending machine 100.Presence of a consumer may be detected by a proximity sensor 135 (see,e.g., FIG. 2 ).

External camera 130 may be configured to capture an image or video ofone or more consumers, and/or an image or video of a product removedfrom cabinet 110. In some embodiments, external camera 130 may beconfigured to identify one or more consumers. Captured images or videomay be used for facial recognition of consumers. In some embodiments,external camera 130 may be configured to identify a hand of a consumerand movements thereof.

External camera 130 may capture images or videos that may be analyzed todetermine a visual characteristic of a product removed from cabinet 110,similar to the operation of internal cameras 120. Thus, a productremoved from cabinet 110 may be detected by external cameras 130 todetermine a shape, size, or coloring of the product. Data from externalcamera 130 may be used to confirm the identity of a product removed fromcabinet 110 as determined by other cameras or sensors described herein.

In some embodiments, external cameras 130 may also be used to determinewhether a consumer is engaging in an unauthorized activity. In someembodiments, external camera 130 may be monitored remotely by anoperator, such as security personnel. In some embodiments, data fromexternal camera 130 may be analyzed by artificial intelligence that ispre-trained to detect unauthorized activities. Artificial intelligencemay detect gestures or movements of a consumer, or the consumer's hands.For example, artificial intelligence may be programmed to detect aconsumer striking the vending machine 100, or blocking a camera 120 toinhibit product identification, among other activities. If unauthorizedactivities are detected, authorities, such as a security personnel orlocal authorities may be alerted. Further, a door of the cabinet may belocked so door cannot be opened and products within cabinet can nolonger be accessed by the consumer. In some embodiments, an alarm 178may be activated if unauthorized activity is detected (see, e.g., FIG. 2).

In some embodiments, the identity of a product removed from cabinet maybe determined using one or more of internal cameras 120, weight sensors140, optical sensors 160, identifier sensor 150, or external cameras 130as described herein. While the identity of a product removed fromcabinet 110 may be determined based on data from internal cameras 120and identifier sensor 150, one or more of an optical sensor 160, weightsensor 140, and external camera 130 may be used to confirm that theidentity is correct. The additional sensors or cameras may also serve asa back-up in the event that a camera or identifier fails to operatecorrectly.

In one example of a product identification, a camera may capture animage of the product to identify the shape of a product (e.g., abottle-shape). However, the shape of the product may correspond tomultiple possible product identities (e.g., Pepsi, diet Pepsi, or cherryPepsi). An identifier sensor may detect an identifier on the product,such as text (e.g., diet Pepsi), which may correspond to multipleproduct identities (e.g., a can or a bottle). Thus, in combination, thedata from the camera and identifier sensor may be analyzed to determinea predicted identity of the product (e.g., a bottle of diet Pepsi). Theanalysis may limit potential product identifications to products in theproduct inventory. Additional data may be collected to confirm that theproduct identification is correct. For example, a weight sensor maydetermine a calculated weight of the product, and the calculated weightmay correspond to a bottle of diet Pepsi, confirming the productidentification. An optical sensor may indicate that the product wasremoved from a location in the cabinet at which a bottle of diet Pepsiwas stored. Further, an external camera may detect a visualcharacteristic, such as the bottle shape once the product is removedfrom the cabinet. Thus, the additional cameras and sensors may help toconfirm that a product has been correctly identified.

In another example, an internal camera may capture images before andafter a consumer removes a product from the cabinet. The images may beanalyzed to determine a location in the cabinet from which the productwas removed. The data from the camera indicating the location of theproduct may be analyzed using a digital map of products in the cabinetto determine the identity of the product at that location. To confirmthe identity of the product, a camera that detects products entering orexiting the cabinet may determine a visual characteristic of the productremoved from the cabinet to confirm the identity of the productdetermination based on the digital map. Alternatively, an identifiersensor may detect an identifier of the product removed from the cabinet.

In some embodiments, artificial intelligence may determine a confidencelevel for product identification based on the cameras or sensors. Sensorfusion algorithms may determine the product identity based on theconfidence level in the identification made by each camera or sensor. Ifthe data is in agreement, the product identity is confirmed. Forexample, if a first camera determines a product removed is Product Awith 80% confidence, a second camera determines the product removed isProduct B with 30% confidence, the algorithm may determine that ProductA is the correct product identification due to the higher confidencelevel. In some embodiments, data from a particular camera or sensor mayhave a stronger weight in determining the identity. In some embodiments,if the confidence level is below a predetermined threshold, e.g., 30%,the data may be disregarded. In some embodiments, if the confidencelevel is below a predetermined threshold, an alert may be sent for anaudit or review to be conducted.

FIG. 15 illustrates an exemplary computer system 1500 in whichembodiments, or portions thereof, may be implemented ascomputer-readable code. Control unit 180 as discussed herein may becomputer systems having all or some of the components of computer system1500 for implementing processes discussed herein.

If programmable logic is used, such logic may execute on a commerciallyavailable processing platform or a special purpose device. One ofordinary skill in the art may appreciate that embodiments of thedisclosed subject matter can be practiced with various computer systemconfigurations, including multi-core multiprocessor systems,minicomputers, and mainframe computers, computer linked or clusteredwith distributed functions, as well as pervasive or miniature computersthat may be embedded into virtually any device.

For instance, at least one processor device and a memory may be used toimplement the above described embodiments. A processor device may be asingle processor, a plurality of processors, or combinations thereof.Processor devices may have one or more processor “cores.”

Various embodiments of the invention(s) may be implemented in terms ofthis example computer system 1500. After reading this description, itwill become apparent to a person skilled in the relevant art how toimplement one or more of the invention(s) using other computer systemsand/or computer architectures. Although operations may be described as asequential process, some of the operations may in fact be performed inparallel, concurrently, and/or in a distributed environment, and withprogram code stored locally or remotely for access by single ormulti-processor machines. In some embodiments, edge computing, cloudcomputing, or a combination thereof may be used. In addition, in someembodiments the order of operations may be rearranged without departingfrom the spirit of the disclosed subject matter.

Processor device 1504 may be a special purpose or a general purposeprocessor device. As will be appreciated by persons skilled in therelevant art, processor device 1504 may also be a single processor in amulti-core/multiprocessor system, such system operating alone, or in acluster of computing devices operating in a cluster or server farm.Processor device 1504 is connected to a communication infrastructure1506, for example, a bus, message queue, network, or multi-coremessage-passing scheme.

Computer system 1500 also includes a main memory 1508, for example,random access memory (RAM), and may also include a secondary memory1510. Secondary memory 1510 may include, for example, a hard disk drive1512, or removable storage drive 1514. Removable storage drive 1514 mayinclude a floppy disk drive, a magnetic tape drive, an optical diskdrive, a flash memory, or the like. The removable storage drive 1514reads from and/or writes to a removable storage unit 1518 in awell-known manner. Removable storage unit 1518 may include a floppydisk, magnetic tape, optical disk, a universal serial bus (USB) drive,etc. which is read by and written to by removable storage drive 1514. Aswill be appreciated by persons skilled in the relevant art, removablestorage unit 1518 includes a computer usable storage medium havingstored therein computer software and/or data.

Computer system 1500 (optionally) includes a display interface 1502(which can include input and output devices such as keyboards, mice,etc.) that forwards graphics, text, and other data from communicationinfrastructure 1506 (or from a frame buffer not shown) for display ondisplay unit 1530.

In alternative implementations, secondary memory 1510 may include othersimilar means for allowing computer programs or other instructions to beloaded into computer system 1500. Such means may include, for example, aremovable storage unit 1522 and an interface 1520. Examples of suchmeans may include a program cartridge and cartridge interface (such asthat found in video game devices), a removable memory chip (such as anEPROM, or PROM) and associated socket, and other removable storage units1522 and interfaces 1520 which allow software and data to be transferredfrom the removable storage unit 1522 to computer system 1500.

Computer system 1500 may also include a communication interface 1524.Communication interface 1524 allows software and data to be transferredbetween computer system 1500 and external devices. Communicationinterface 1524 may include a modem, a network interface (such as anEthernet card), a communication port, a PCMCIA slot and card, or thelike. Software and data transferred via communication interface 1524 maybe in the form of signals, which may be electronic, electromagnetic,optical, or other signals capable of being received by communicationinterface 1524. These signals may be provided to communication interface1524 via a communication path 1526. Communication path 1526 carriessignals and may be implemented using wire or cable, fiber optics, aphone line, a cellular phone link, an RF link or other communicationchannels.

In this document, the terms “computer program medium” and “computerusable medium” are used to generally refer to media such as removablestorage unit 1518, removable storage unit 1522, and a hard diskinstalled in hard disk drive 1512. Computer program medium and computerusable medium may also refer to memories, such as main memory 1508 andsecondary memory 1510, which may be memory semiconductors (e.g. DRAMs,etc.).

Computer programs (also called computer control logic) are stored inmain memory 1508 and/or secondary memory 1510. Computer programs mayalso be received via communication interface 1524. Such computerprograms, when executed, enable computer system 1500 to implement theembodiments as discussed herein. In particular, the computer programs,when executed, enable processor device 1504 to implement the processesof the embodiments discussed here. Accordingly, such computer programsrepresent controllers of the computer system 1500. Where the embodimentsare implemented using software, the software may be stored in a computerprogram product and loaded into computer system 1500 using removablestorage drive 1514, interface 1520, and hard disk drive 1512, orcommunication interface 1524.

Embodiments of the invention(s) also may be directed to computer programproducts comprising software stored on any computer useable medium. Suchsoftware, when executed in one or more data processing device, causes adata processing device(s) to operate as described herein. Embodiments ofthe invention(s) may employ any computer useable or readable medium.Examples of computer useable mediums include, but are not limited to,primary storage devices (e.g., any type of random access memory),secondary storage devices (e.g., hard drives, floppy disks, CD ROMS, ZIPdisks, tapes, magnetic storage devices, and optical storage devices,MEMS, nanotechnological storage device, etc.).

It is to be appreciated that the Detailed Description section, and notthe Summary and Abstract sections, is intended to be used to interpretthe claims. The Summary and Abstract sections may set forth one or morebut not all exemplary embodiments of the present invention(s) ascontemplated by the inventors, and thus, are not intended to limit thepresent invention(s) and the appended claims in any way.

The present invention has been described above with the aid offunctional building blocks illustrating the implementation of specifiedfunctions and relationships thereof. The boundaries of these functionalbuilding blocks have been arbitrarily defined herein for the convenienceof the description. Alternate boundaries can be defined so long as thespecified functions and relationships thereof are appropriatelyperformed.

The foregoing description of the specific embodiments will so fullyreveal the general nature of the invention(s) that others can, byapplying knowledge within the skill of the art, readily modify and/oradapt for various applications such specific embodiments, without undueexperimentation, and without departing from the general concept of thepresent invention(s). Therefore, such adaptations and modifications areintended to be within the meaning and range of equivalents of thedisclosed embodiments, based on the teaching and guidance presentedherein. It is to be understood that the phraseology or terminologyherein is for the purpose of description and not of limitation, suchthat the terminology or phraseology of the present specification is tobe interpreted by the skilled artisan in light of the teachings andguidance herein.

The breadth and scope of the present invention(s) should not be limitedby any of the above-described exemplary embodiments, but should bedefined only in accordance with the following claims and theirequivalents.

What is claimed is:
 1. A vending system, comprising: a cabinetcomprising a storage area for storing products, and a door movablycoupled to the cabinet, wherein the door is movable from a closedconfiguration to an open configuration in which the products areaccessible to a consumer; a camera arranged within the cabinet andconfigured to detect one or more visual characteristics of a productremoved from the cabinet when the door is in the open configuration; anidentifier sensor configured to detect an identifier of the productremoved from the cabinet; and a control unit in communication with thecamera and with the identifier sensor, wherein the control unit isconfigured to determine an identity the product removed from the cabinetbased at least in part on a comparison of the visual characteristic andthe identifier to a database of product information.
 2. The vendingsystem of claim 1, wherein the camera is one of a plurality of camerasarranged within the cabinet.
 3. The vending system of claim 2, whereinthe plurality of cameras are arranged at a perimeter of a front portionof the cabinet.
 4. The vending system of claim 1, wherein the one ormore visual characteristics comprises a shape of the product, adimension of the product, or a coloring of the product.
 5. The vendingsystem of claim 1, wherein the identifier sensor is the camera.
 6. Thevending system of claim 1, wherein the identifier sensor is configuredto detect an identifier that is a barcode or a QR code.
 7. The vendingsystem of claim 1, further comprising a weight sensor configured todetermine a weight of the product removed from the cabinet.
 8. Thevending system of claim 1, further comprising an external cameraarranged at a location outside of the cabinet and configured to detect asecond visual characteristic of the product removed from the cabinet. 9.The vending system of claim 1, further comprising a payment processingunit configured to receive a payment, wherein the control unit isconfigured to unlock the door when the payment is received.
 10. Thevending system of claim 1, wherein the database comprises a list ofproducts and corresponding visual characteristics.
 11. The vendingsystem of claim 1, wherein the control unit is configured to determinethe identity of the product using a machine learning algorithm.
 12. Avending system, comprising: a cabinet comprising a storage area forstoring products, and a door movably coupled to the cabinet, wherein thedoor is movable from a closed configuration to an open configuration inwhich the products are accessible to a consumer; an internal cameraarranged within the cabinet configured to capture a first image of theproducts prior to a consumer accessing the storage area and to capture asecond image of the plurality of products after the consumer accessesthe cabinet; an identifier sensor configured to detect an identifier ofthe product removed from the cabinet; and a control unit incommunication with the internal camera and the identifier sensor,wherein the control unit is configured to determine an identity of theproduct removed from the cabinet by analyzing the first image and thesecond image, and wherein the control unit is further configured toconfirm the identity of the product removed from the cabinet based onthe identifier of the product removed from the cabinet as determined bythe identifier sensor.
 13. The vending system of claim 12, furthercomprising a weight sensor configured to determine a weight of theproduct removed from the cabinet.
 14. The vending system of claim 12,further comprising an external camera arranged at a location outside ofthe cabinet and configured to detect a second visual characteristic ofthe product removed from the cabinet.
 15. The vending system of claim12, further comprising an optical sensor configured to detect datarelating to the product removed from the cabinet.
 16. The vending systemof claim 15, wherein the optical sensor comprises an RFID sensor, aLIDAR sensor, or an MRI sensor.
 17. A vending system, comprising: acabinet comprising a storage area for storing products; at least one ofa camera or a sensor arranged within the cabinet and configured todetect a location from which a product was removed from the cabinet andto detect a visual characteristic of the product removed from thecabinet; and a control unit in communication with the at least one ofthe camera or the sensor, wherein the control unit is configured togenerate a digital map using data from the at least one of the camera orthe sensor, wherein the digital map comprises a location and an identityof each of the products within the storage area of the cabinet, whereinthe control unit is configured to determine an identity of the productremoved from the cabinet based at least in part on the detected visualcharacteristic, and wherein the control unit is configured to confirmthe identity of the product based on the digital map.
 18. The vendingsystem of claim 17, further comprising an identifier sensor configuredto detect an identifier of the product removed from the cabinet.
 19. Thevending system of claim 17, further comprising an optical sensorconfigured to detect data relating to the product removed from thecabinet.
 20. The vending system of claim 17, wherein the control unit isconfigured to determine the identity of the product removed from thecabinet using a machine learning model.